Phase 9 — Crawl and Index Health, and Enhanced Testing

Markers: ▶ being worked on, ⚑ implemented and waiting to be reviewed, ✓ done, ✓✗ closed by deciding not to do it, ? open decision, unmarked = planned.

1. Crawling using synthetic data. A crawl of any size can be run on one machine without fetching anything from the web. The pages are made up as the crawl asks for them, so a run repeats exactly and costs nobody bandwidth.
   1. ⚑ A synthetic crawl tab is added to the crawl options. Manage Crawl, then Edit Crawl Options, gains a tab beside the web crawl one. It keeps the settings that still mean something for a made-up web and replaces the ones that do not.
      1. ⚑ The settings about reaching the live web are dropped. Repeat type, crawl sleep, robots.txt, restrict sites by url, disallowed sites and seed sites all describe fetching from real hosts. A made-up web has no hosts to be polite to and no addresses to restrict.
      2. ⚑ The shape and size of the made-up web are chosen. A graph type says how the pages link to one another. A graph size says how many pages the web holds. A third setting says which pages the crawl starts from, taking the place of seed sites.
      3. ⚑ What a page is and what it says are chosen by chance. One model gives the chance a page is html, pdf or an image. Another gives the words that appear on it. The same seed gives the same web every run, so a result can be compared with the one before it.
      4. The settings suit both a small check and a large load. A small web makes it possible to ask whether every page indexed can be recalled and whether results come back in the order expected. A large one puts the web server, the queue server and the fetchers under load worth measuring.
   2. ⚑ A synthetic crawl makes its own pages rather than fetching them. Where that tab is the one chosen, the fetcher takes its pages from a function that generates them in place of FetchUrl::getPages. Everything after the fetch runs unchanged, so what is measured is the real crawl. A page is a tuple of numbers: a grid page steps one along in each component, a tree page appends each child, a power law page draws its targets by hash. By domain the tuple is one hyphened label, http://i-j.web/, so each page is its own company level domain; by path it sits under http://synth.web/i/j/. The queue server and fetcher see a plain web crawl; only the fetch call differs. Two rules of Yioop's had to give: a host with no dot has no company level domain and is dropped, and five consonants in a row read as spam, so y is now a vowel there. Run as daemons with errors shown: a grid by path reached 66 pages on one robots.txt, by domain 25 pages on one robots.txt per page, and a search of the finished index found a made-up word.
2. The slowdown over a long crawl is found and fixed. A synthetic crawl is run long enough for the time per page to climb. Where it starts climbing says which stage is at fault, and what that reading names is what gets fixed.
3. An index is never left corrupt by a crawl that stops part way through. Phase 4 added guards that keep a corrupt index from taking the server down. Those are guards rather than prevention. This item is the prevention.
   1. Every point that could leave a half-written entry is named. The walk covers the partition append, the dictionary merge and the postings write. What is looked for is a write that a later read takes as a garbage length or offset.
   2. A write is either finished or undone, never left part way. A write goes to a new name and is put in place at the end, or a small mark says a step was begun. On starting again the mark says whether to finish the step or drop it.
   3. The recovery is shown by stopping a crawl on purpose. A synthetic crawl is driven and the queue server is stopped at chosen points in the write path. The index is read back afterward and must come back clean every time.
4. Other work, grouped by what it touched. Each is a deliverable of its own that came up during the arc. They are grouped by the part of Yioop they concern rather than listed one by one.
   1. A large download no longer holds the web server or its memory. One reader pulling a large video stalled every other request for as long as the download took. A three hundred megabyte video ended the process outright.
      1. A large file is drained by the event loop rather than by the request. A resource request with no byte range is now deferred, and the whole file case names the file instead of reading it out. Measured on a two hundred megabyte file: pages answered in twenty seconds before and under five milliseconds after.
      2. A request keeps its streaming context while it waits. The context saying which protocol a reply goes out on was not saved with the rest of a waiting request. A resumed request therefore gathered its whole body in memory. It is saved and restored with the other request state.
      3. ⚑ A large download over either protocol arrives whole again. A ten megabyte pdf came back as headers and no body. Three faults on the deferred file path: a resumed request found no task to hand its file to, since the task was cleared when the route returned; the HTTP/1.1 headers were finished from an empty string and said a length of zero; and a stream whose buffer had drained was closed while its file still waited. The fiber now sets its own task on entry, the file's size is passed to the headers, and a stream with a file pending is not finished. Measured over TLS: both protocols deliver 10,485,755 bytes identical to the upload; a byte range and a small file still work.
      4. One block of IP addresses keeps one share of the link. Files going to the same block divide one turn of the write loop between them. Several downloads from one place cannot take several shares. Traffic between Yioop machines keeps a whole turn each.
   2. A document thumb on a wiki page shows a picture, not a frame. A resource-thumb mark for a pdf drew an iframe of the document where no thumbnail was there. The mark is now always a link with a picture, the thumbnail is made from the document on the first read that asks for it, and an image thumb draws the thumbnail address rather than the full picture.
   3. ⚑ A reader who may not edit, or see a private history or source, is answered 403. A stranger's arg=edit with a stale token was quietly downgraded to a page read and served 200; a private page's source was served at 200 too. The group controller now answers 403 for a writing arg on a request whose token is bad; the wiki component refuses edit, history and source by the group's Access setting, where only Read Write Wiki lets a member edit. Walked as a stranger, a member of a Read group, a member of a Read Write Wiki group, and root.
   4. ⚑ The server log names the signed in user, marks each reply's speed, and names a timer that held the loop. Each finished-reply line carries the username after the address, and its time ends in -slow, -slower or -slowest by band so one grep finds a band. A timer callback that runs past a tenth of a second writes a line with its length and the file and line it was set at, since a request arriving during one waits the whole of it and two cheap 403s that took 2.6 seconds in the same second had waited on one such callback.
   5. ⚑ Every request's completion line in the server log says how long it took. The line that records a reply's status and size ran only for an immediate HTTP/1 reply, so no HTTP/2 page and no deferred request, which is every search, ever logged one. It now runs once on every reply path, a deferred reply logs after its own environment is applied with a file reply's true size, a redirect logs as 302, and the line ends with the milliseconds since the request arrived. Measured here: a search over h2 22ms, the same page over h1 3ms, a ten megabyte download 2ms to hand off.
   6. ⚑ A pdf thumbnail draws a magazine cover as a viewer does. Two covers came out scrambled or blank. Six faults in the page reader, each found against a pdftoppm render of the same page.
      1. A large four ink jpeg keeps its black ink. Past four million points the picture fell to the drawing library, which drops the black ink. Such a picture is now read one shade per block, an eighth the size, in half a second.
      2. A picture is laid through its soft mask. A figure paired with a gray mask was laid opaque over the cover. The mask now sets each point's transparency.
      3. A composite font is read and its letters reached by number. Such a font keeps its descriptor on a descendant and names each letter by a two byte place in the font. The reader follows the descendant, reads the width table, and walks a run two bytes at a time.
      4. A run keeps its zero bytes and its saved color. Trimming a run cut the zero byte that begins a low numbered letter, and a color set inside a saved state was read on past the restore. Only blank space is trimmed, and the color is saved and restored with the frame.
      5. Letters are set as far apart as the page says. The character spacing operator is read and carried on each run.
      6. A picture named inside a form the page draws is one of the page's pictures. A cover that was one photograph inside one form came out blank. The resources of each form the page draws join the page's own.